98 research outputs found
Ultra-high Dimensional Multiple Output Learning With Simultaneous Orthogonal Matching Pursuit: A Sure Screening Approach
We propose a novel application of the Simultaneous Orthogonal Matching
Pursuit (S-OMP) procedure for sparsistant variable selection in ultra-high
dimensional multi-task regression problems. Screening of variables, as
introduced in \cite{fan08sis}, is an efficient and highly scalable way to
remove many irrelevant variables from the set of all variables, while retaining
all the relevant variables. S-OMP can be applied to problems with hundreds of
thousands of variables and once the number of variables is reduced to a
manageable size, a more computationally demanding procedure can be used to
identify the relevant variables for each of the regression outputs. To our
knowledge, this is the first attempt to utilize relatedness of multiple outputs
to perform fast screening of relevant variables. As our main theoretical
contribution, we prove that, asymptotically, S-OMP is guaranteed to reduce an
ultra-high number of variables to below the sample size without losing true
relevant variables. We also provide formal evidence that a modified Bayesian
information criterion (BIC) can be used to efficiently determine the number of
iterations in S-OMP. We further provide empirical evidence on the benefit of
variable selection using multiple regression outputs jointly, as opposed to
performing variable selection for each output separately. The finite sample
performance of S-OMP is demonstrated on extensive simulation studies, and on a
genetic association mapping problem. Adaptive Lasso; Greedy forward
regression; Orthogonal matching pursuit; Multi-output regression; Multi-task
learning; Simultaneous orthogonal matching pursuit; Sure screening; Variable
selectio
Scalable Peaceman-Rachford Splitting Method with Proximal Terms
Along with developing of Peaceman-Rachford Splittling Method (PRSM), many
batch algorithms based on it have been studied very deeply. But almost no
algorithm focused on the performance of stochastic version of PRSM. In this
paper, we propose a new stochastic algorithm based on PRSM, prove its
convergence rate in ergodic sense, and test its performance on both artificial
and real data. We show that our proposed algorithm, Stochastic Scalable PRSM
(SS-PRSM), enjoys the convergence rate, which is the same as those
newest stochastic algorithms that based on ADMM but faster than general
Stochastic ADMM (which is ). Our algorithm also owns wide
flexibility, outperforms many state-of-the-art stochastic algorithms coming
from ADMM, and has low memory cost in large-scale splitting optimization
problems
- …